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MANUAL TOOLS FOR MODEL BASED IMAGE SEGMENTATION 

DESCRIPTION 

The present invention relates to the diagnostic imaging systems and 
methods. It finds particular application in conjunction with the model based image 
segmentation of diagnostic medical images and will be described with particular reference 
thereto. Although described by the way of example with reference to CT tomography, it 
5 will further be appreciated that the invention is equally applicable to other diagnostic 
imaging techniques which generate 3D image representations. 

Radiation therapy has been recently experiencing a transition from 
conformal methods to Intensity Modulation Radiation Therapy (EMRP). IMRT enables an 
improved dose distribution in the patient's body and makes possible precise delivery of 
10 high radiation dose directly to the tumor while maximally sparing the surrounding healthy 
tissue. Accurate target and "organ at risk" delineation is important in IMRT. Presently, the 
procedure is performed manually in 2D slices, which is cumbersome and the most time- 
consuming part of the radiation therapy planning process. The use of robust and reliable 
automatic segmentation technique might substantially facilitate the planning process and 
1 5 increase patient throughput. 

Model based image segmentation is a process of segmenting (contouring) 
medical diagnostic images that is used to improve robustness of segmentation methods. 
Typically, a pre-determined 3D model of the region of interest or organ to be segmented 
in the diagnostic image is selected. The model represents an anatomical organ such as a 
20 bladder or femur, but it may also represent a structure such as a target volume for 
radiotherapy. In many cases, the model can be used to aid automated image segmentation 
by providing knowledge of the organ shape as an initial starting point for the automated 
segmentation process. However, in some instances, the auto-segmentation of the image 
may not be possible, or it is not robust enough to fit a specific organ or a section to the 
25 model accurately. Particularly, application of the auto-segmentation to the image data is 
difficult due to insufficient soft tissue contrast in CT data, high organ variability, and 
image artifacts, e.g. caused by dental fillings or metal implants. It would be desirable to 
be able to initiate the segmentation with a model and further complete an accurate 
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segmentation when auto-segmentation is not practical or to enhance the auto- 
segmentation result for specific situations after auto-segmentation has been completed. 

There is a need for the method and apparatus to provide the image 
segmentation of the model based image that is easily adapted to match a specific patient's 
5 anatomy. The present invention provides a new and improved imaging apparatus and 
method which overcomes the above-referenced problems and others. 

In accordance with one aspect of the present invention, a diagnostic 
10 imaging system is disclosed. A means selects a shape model of an organ. A means best 

fits the selected model to an image data. A manual means modifies selected regions of 

the model to precisely match the image data. 

In accordance with another aspect of the present invention, a method of 

segmenting an image of a diagnostic imaging system is disclosed. A shape model of an 
15 organ is selected. The selected model is dragged and dropped on an image data. The 

selected model is globally scaled, rotated and translated to best fit the image data. Local 

regions of the model are modified with a set of manual tools to precisely match the image 

data. 

One advantage of the present invention resides in enabling the manipulation 
20 of the models to match subject's anatomy. 

Another advantage resides in providing a set of diagnostic image 
modification tools enable the user to modify the models with a mouse. 

Still further advantages and benefits of the present invention will become 
apparent to those of ordinary skill in the art upon reading and understanding the following 
25 detailed description of the preferred embodiments . 

The invention may take form in various components and arrangements of 
components, and in various steps and arrangements of steps. The drawings are only for 
30 purposes of illustrating the preferred embodiments and are not to be construed as limiting 
the invention. 

FIGURE 1 is a diagrammatic illustration of a diagnostic imaging system; 
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FIGURE 2 is a graphical representation of an organ model using triangular 
surfaces for use with aspects of the present invention; 

FIGURES 3-4 are graphical representations of a Gaussian pull tool in 
accordance with the present invention; 
5 FIGURES 5-6 are graphical representations of a Sphere push tool in 

accordance with the present invention; 

FIGURES 7-9 are graphical representations of a Pencil tool in accordance 
with the present invention. 



10 

With reference to FIGURE 1, an operation of an imaging system 10 is 
controlled from an operator workstation 12 which includes a hardware means 14 and a 
software means 16 for carrying out the necessary image processing functions and 
operations. Typically, the imaging system 10 includes a diagnostic imager such as CT 

15 scanner 18 including a non-rotating gantry 20. An x-ray tube 22 is mounted to a rotating 
gantry 24. A bore 26 defines an examination region of the CT scanner 18. An array of 
radiation detectors 28 is disposed on the rotating gantry 24 to receive radiation from the x- 
ray tube 22 after the x-rays transverse the examination region 26. Alternatively, the array 
of detectors 28 may be positioned on the non-rotating gantry 20. 

20 Typically, the imaging technician performs a scan using the workstation 12. 

Diagnostic data from the scanner 18 is reconstructed by a reconstruction processor 30 into 
3D electronic image representations which are stored in a diagnostic image memory 32. 
The reconstruction processor 30 may be incorporated into the workstation 12, the scanner 
18, or may be a shared resource among a plurality of scanners and workstations. The 

25 diagnostic image memory 32 preferably stores a three-dimensional image representation of 
an examined region of the subject. A video processor 34 converts selected portions of the 
three-dimensional image representation into appropriate format for display on a video 
monitor 36. The operator provides input to the workstation 12 by using an operator input 
device 38, such as a mouse, touch screen, touch pad, keyboard, or other device. 

30 With continuing reference to FIGURE 1 and further reference to FIGURE 

2, an organ model database 50 stores predetermined models 52 of specific organs and 
general shapes of areas of interest that could correspond to radiotherapy treatment areas of 
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interest, e.g., shapes approximating a tumor shape to be treated with radiation. Typically, 
the organ models 52 are defined as a set of polygons describing a surface. Preferably, the 
polygons are triangles and represented by a flexible triangular mesh 54. The basic 
structure is a list of vertices in (x, y, z) coordinates and a list of polygons which identify 
5 the vertices which comprise each polygon, e.g. each triangle has three vertices. Storage of 
basic structures and automatic segmentation of images using such triangular structures is 
more fully described in pending U.S. Patent Application No. 10/091,049 having 
Publication No. 2002/0182466 Al entitled Image Segmentation by Weese, et al. 

With continuing reference to FIGURE 1, the user selects a model of an 

10 organ from the organ model database 50 via a model selecting means 60. Preferably, the 
software 16 includes a user interface means which allow the user to select models by 
dragging and dropping the organ model over the subject anatomy represented by the image 
data while watching a superimposition of the diagnostic image and the organ model on the 
monitor 36. Various displays are contemplated. In one advantageous display format, three 

15 orthogonal slices through the 3D image which intersect at a common point are displayed 
concurrently in quadrants of the monitor. By shifting the intersection point and/or dragging 
the organ model, the fit in all three dimensions is readily checked. 

The user best fits the model to the organ using of a set of global tools 62 
which apply transformation to the entire model on the image. The global tools 62 include 

20 rotation, translation and scaling tools that allow the user to rotate, translate, and scale the 
model. The global tools 62 are applied by a use of the mouse 38 on each (x, y, z) 
dimension of the model, e.g. the mouse motion is converted into translation, scale or 
rotation such that all vertices in the model are transformed by the defined translation, scale, 
or rotation. 

25 An auto-segmentation means or process 64 automatically adapts the best 

fitted model to the boundaries of the anatomical structures of interest. By sliding the 
intersection point, the user can check the fit in various directions and slices. If the user 
determines that the results of the auto-segmentation process 64 are not satisfactory, e.g. the 
desired segmentation accuracy is not achieved, the user initiates image modification via an 

30 image modification means 66 which includes a set of manual local tools 68 which allows 
the user to manipulate local regions of the model 52 to match the image data more 
accurately or in accordance with user's preferences. Alternatively, when the user 
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determines that the auto-segmentation is not possible, the auto-segmentation process 64 is 
skipped. The local tools 68 comprise three main functions: selection of the local region 
(vertices) to be modified, the method by which the vertices are transformed, and the 
translation of the mouse motion into parameters defining the transformation. 
5 The selection of the vertices is based either on the distance from the mouse 

position or the layers of vertex neighbors from the closest vertex to the mouse location. 
In the first case, all vertices within a specified distance from the mouse are selected. In 
the latter case, the vertex closest to the mouse is selected. All vertices which share a 
triangle with the first vertex are considered neighbors and comprise the first neighbor 

10 layer. A second neighbor layer is all vertices which share a triangle with any of the first 
layer of vertices. In this case, the selection of the vertices to be deformed is based on the 
number of neighbor layers to be used. 

Additionally, control parameters related to local manipulation tools are 
stored as part of the organ model. In this way, optimal tool settings are maintained as part 

15 of the organ model. Of course, it is also contemplated that the manual tools (68) may be 
used to manipulate boundaries between multiple organs at one time or within a regional 
area with a single mouse motion. 

The image undergoing segmentation and segmented images are stored in a 
data memory 70. 

20 With continuing reference to FIGURE 1 and further reference to 

FIGURES 3-4, a Gaussian pull tool 72 deforms the organ model by pulling the local 
vertices by a Gaussian weighted distance of the mouse motion d. Thus, the vertex that is 
at the initial position 74 of the mouse moves into position 76 the same distance d as the 
mouse motion d. Vertices farther away from the mouse move a shorter distance based on 

25 a Gaussian function scaling of the mouse motion. Typically, the Gaussian tool 72 is 
controlled by a single Gaussian radius which defines the width of the Gaussian spread. 
Alternatively, the Gaussian tool 72 is controlled by separate x- and y-Gaussian radii 
which allow for the x-radius to be used in the plane of motion of the mouse, and the y- 
radius to be used orthogonally to the drawing plane. In another embodiment, the 

30 Gaussian tool 72 is controlled by a function, e.g. triangle, parabola, etc., that smoothly 
transitions from 1 to 0 with the appropriate set of parameters to accomplish a 
transformation of the selected vertices. 
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In one embodiment, the Gaussian pull tool 72 pulls a Gaussian shaped 
distortion (or other functional shape the smoothly transitions from 1 to 0) but derives the 
distance that the distortion is pulled from the distance of the mouse position from the 
organ model. The organ model 52 is pulled directly to the mouse position enabling 
5 smooth drawing, rather than having to click up and down on the mouse to grab and stretch 
the organ. 

With continuing reference to FIGURE 1 and further reference to 
FIGURES 5-6, a sphere push tool 80 searches for all vertices contained within a sphere 
82 of a specified radius R around the mouse location 84. Each vertex in the sphere 82 is 

10 moved to the surface of the sphere along the vector 86 from the mouse location 84 
through the original vertex location. As the mouse moves the push tool 80 by moving 
location 84, the organ model 52 is pushed either inward or outward depending on the 
location of the vertex with respect to the mouse location 84. The Sphere tool 80 is 
controlled by a single sphere radius parameter that is preferably stored with the individual 

15 organ model. In this way, the surface is deformed analogous to pressing a spherical tool 
of the selected radius against a soft clay surface. But, on the computer, the tool 80 can be 
placed inside the model to push out or outside to push in. Of course, other surfaces of 
predetermined shapes such as ellipses are also contemplated. Optionally, the model 
surface can be re-triangulated after the surface modification to smooth the reshaped organ 

20 surface. 

With continuing reference to FIGURE 1 and further reference to 
FIGURES 7-9, a pencil draw tool 90 is used to deform an original boundary 92 of the 
organ model such that it aligns with a drawing path or actual boundary 94 of the drawing 
motion of the mouse. While the user uses the mouse to draw a line along the actual 
25 boundary 94 of the structure to be segmented, the model's original boundary 92 deforms 
to match the drawing path 94 in the plane of mouse motion. Outside of the plane of 
mouse motion, the organ model deforms to perform a smooth transition in the model. 
Rather than drawing a complete path 94, the drawing path 94 may be approximated with a 
series of dots. 

30 The Pencil draw tool 90 recognizes begin 96 and end 98 points of each 

mouse step and defines a capture plane 100 through a vector whose normal vector is in 
the plane of the mouse motion and is normal to the mouse motion direction. Two end 
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planes 102, 104, which are defined at the start and end points 96, 98, identify a capture 
range 106 around the mouse motion vector. Vertices located within the capture range 106 
are pulled towards the capture plane 100. Vertices that lie on the plane 100 are pulled 
onto the plane 100. Vertices that lie further from the mouse motion plane are pulled with 
5 a Gaussian weighting of the distance to the capture plane 100 based on the distance from 
the mouse motion plane. 

In one embodiment, the Pencil tool 90 is used to shrink fit an organ model 
to a predefined set of contours for a particular organ where the mouse motion is replaced 
with successive vertices of the pre-defined contour. 
10 Preferably, the Pencil draw tool 90 is controlled by a In-Draw Plane 

distance which defines the maximum distance between a vertex of the organ model and 
the mouse for the vertex to be captured by the Pencil tool 90, and a From-Draw Plane 
parameter which dictates how the model 52 is deformed in the direction orthogonal to the 
drawing plane and represents the width of the Gaussian function used to weight the 
15 distance that the vertices move. In one embodiment, the Pencil draw tool 90 is controlled 
by a function that smoothly transitions from 1 to 0 to perform the weighting of the 
distance of vertex motion for vertices that do not lie on the drawing plane. 

Optionally, the auto-segmentation process 64 is run after manual 
segmentation, preferably freezing the manually adjusted model surfaces against further 
20 modification or modification beyond preselected criteria. 

The invention has been described with reference to the preferred 
embodiments. Modifications and alterations may occur to others upon a reading and 
understanding of the preceding detailed description. It is intended that the invention be 
constructed as including all such modifications and alterations insofar as they come 
25 within the scope of the appended claims or the equivalents thereof. 
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